Method and system for determining abnormality in medical device

ABSTRACT

A method for determining an abnormality in a medical device from a medical image is provided. The method for determining an abnormality in a medical device comprises receiving a medical image, and detecting information on at least a part of a target medical device included in the received medical image.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. Application No.17/973,672 filed Oct. 26, 2022, which is a continuation of InternationalPatent Application No. PCT/KR2021/006163, filed May 17, 2021, which isbased upon and claims the benefit of priority to Korean PatentApplication No. 10-2020-0059347, filed on May 18, 2020 and Korean PatentApplication No. 10-2021-0063464, filed on May 17, 2021. The disclosuresof the above-listed applications are hereby incorporated by referenceherein in their entirety.

TECHNICAL FIELD

The present disclosure relates to a method and a system for determiningan abnormality in a medical device, and specifically, to a method and asystem for determining an abnormality in a medical device included in amedical image.

BACKGROUND

In general, a doctor may use a medical image to determine a condition ofa patient. For example, the doctor may check, through the medical image,the position, size, and shape of a lesion of the patient, and also theposition of the medical device inserted into the body of the patient orplaced outside the patient, and the like, and perform necessarydiagnosis and treatment. In particular, for the required diagnosis andtreatment, it is very important to determine whether or not the medicaldevice is in a normal position and the like.

Meanwhile, the normal position of the medical device may be determinedaccording to the type of the medical device, the anatomical position ofthe human body organ, and the like. In other words, the normal positionof the medical device may be determined differently for each patientaccording to the type of the medical device, the position and size ofthe body organs, and the like. Accordingly, it may not be easy for thedoctor to determine the normal position of the medical device based onthe medical image of the patient. In particular, for patients withmultiple complex findings, it may be difficult for the doctor todetermine, based on the medical image, whether or not the medical deviceis in the normal position.

SUMMARY

In order to solve the problems described above, the present disclosureprovides a method, a non-transitory computer-readable recording mediumstoring instructions, and an apparatus (system) for determining anabnormality in a medical device from a medical image.

The present disclosure may be implemented in various ways, including amethod, a system (apparatus), and a computer-readable storage mediumstored with a computer program.

According to an embodiment of the present disclosure, a method fordetermining an abnormality in a medical device in medical image isprovided, which may be executed by at least one processor and includereceiving a medical image, and detecting information on at least a partof a target medical device included in the received medical image.

According to an embodiment of the present disclosure, the detecting mayinclude detecting the information on a position of the at least the partof the target medical device in the received medical image by using afirst machine learning model.

According to an embodiment of the present disclosure, the method mayfurther includes acquiring a plurality of reference medical imagesincluding one or more reference medical devices, and acquiring anannotation for a position of at least a part of the one or morereference medical devices included in the plurality of reference medicalimages. The first machine learning model may be trained to receive theplurality of reference medical images, and detect information on thereference medical devices included in each of the plurality of referencemedical images based on the annotation for the position of at least thepart of the one or more reference medical devices.

According to an embodiment of the present disclosure, the detecting theinformation on the position of the at least the part of the targetmedical device may include determining whether or not the target medicaldevice is included in the received medical image by using a secondmachine learning model, and if the target medical device is included inthe received medical image, detecting the information on the position ofthe at least the part of the target medical device in the receivedmedical image by using the first machine learning model.

According to an embodiment of the present disclosure, the determiningwhether or not the target medical device is included in the receivedmedical image by using the second machine learning model may includedetermining whether or not a medical device included in the receivedmedical image belongs to the same medical device group as the targetmedical device. The second machine learning model is trained to receivea plurality of reference medical images and output a medical devicegroup to which a reference medical device included in each of theplurality of reference medical images belongs.

According to an embodiment of the present disclosure, the detecting mayinclude extracting, from the received medical image, a fiducial markerassociated with the target medical device, and determining presence orabsence of an abnormality in the target medical device based on theinformation on the target medical device and the extracted fiducialmarker.

According to an embodiment of the present disclosure, the extracting mayinclude extracting, from the received medical image, a fiducial markerassociated with the target medical device by using a third machinelearning model.

According to an embodiment of the present disclosure, the method mayfurther include acquiring a plurality of reference medical imagesincluding one or more reference medical devices, and acquiring anannotation for a reference fiducial marker associated with the one ormore reference medical devices included in the plurality of referencemedical images. The third machine learning model may be trained toreceive the plurality of reference medical images, and extract thereference fiducial marker associated with the one or more referencemedical devices in the plurality of reference medical images based onthe annotation for the reference fiducial marker associated with the oneor more reference medical devices.

According to an embodiment of the present disclosure, the determiningmay include determining a normal area of the target medical device basedon the extracted fiducial marker, and determining whether or not the atleast the part of the target medical device is positioned in the normalarea.

According to an embodiment of the present disclosure, a method fordetermining an abnormality in a medical device in medical imaging isprovided, which may be executed by at least one processor and includereceiving a reference medical image, determining a normal areaassociated with a reference medical device in the reference medicalimage, generating a first set of training data in which at least a partof the reference medical device is placed in the determined normal areain the reference medical image, generating a second set of training datain which the at least the part of the reference medical device is placedin an area other than the determined normal area in the referencemedical image, and training a fourth machine learning model fordetermining presence or absence of an abnormality in the referencemedical device based on the first set of training data and the secondset of training data.

According to an embodiment of the present disclosure, the method mayinclude receiving a medical image, and determining presence or absenceof an abnormality in a target medical device included in the medicalimage by using a fourth machine learning model.

According to an embodiment of the present disclosure, the determiningmay include receiving, from an external device, information on thenormal area associated with the position of the at least the part of thereference medical device, and applying a normal area associated with theposition of the at least the part associated with the reference medicaldevice to the reference medical image.

According to an embodiment of the present disclosure, the determiningmay include receiving, from an external device, information on areference medical device for generating the training data, andextracting normal area associated with the reference medical device fromin the reference medical image, based on the received information on thereference medical device and the information on the reference medicalimage.

According to an embodiment of the present disclosure, the fourth machinelearning model includes a binary classification model trained toclassify the reference medical image into normal data or abnormal data.

A non-transitory computer-readable recording medium storing instructionsthat, when executed by one or more processors, cause performance of themethod described above according to the embodiment.

An information processing system according to another embodiment of thepresent disclosure is provided, which may include a memory storing oneor more instructions, and a processor configured to execute the storedone or more instructions to receive a medical image and detectinformation on at least a part of a target medical device included inthe received medical image.

An information processing system according to another embodiment of thepresent disclosure is provided, which may include a memory storing oneor more instructions, and a processor configured to execute the storedone or more instructions to receive a reference medical image, determinea normal area associated with a reference medical device in thereference medical image, generate a first set of training data in whichat least a part of the reference medical device is placed in thedetermined normal area in the reference medical image, generate a secondset of training data in which the at least the part of the referencemedical device is placed in an area other than the determined normalarea in the reference medical image, and train a fourth machine learningmodel for determining presence or absence of an abnormality in thereference medical device based on the first set of training data and thesecond set of training data.

According to some embodiments of the present disclosure, user may easilyacquire information on the presence or absence of an abnormality in themedical device associated with a patient in the medical image, since theinformation on the presence or absence of an abnormality is determinedfrom the medical image. In particular, even in the case of a patientaccompanied by several complex findings, the information on the presenceor absence of an abnormality in the medical device can be acquired fromthe medical image.

According to some embodiments of the present disclosure, after aprocedure of inserting the medical device into a body or attaching it toa body surface of the patient, the user can quickly and accurately checkwhether or not the procedure is performed correctly. Accordingly, if theinserted medical device is incorrectly positioned or the operation isnot performed correctly, the user can take quick action.

According to some embodiments of the present disclosure, even when it isdifficult to collect a large amount of medical images showing themedical device placed normally or abnormally, by generating a pluralityof training medical images, it is possible to effectively train anartificial neural network model for determining the presence or absenceof an abnormality in the medical device.

The effects of the present disclosure are not limited to the effectsdescribed above, and other effects not mentioned will be able to beclearly understood by those of ordinary skill in the art (referred to as“those skilled in the art”) from the description of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentdisclosure will become more apparent to those of ordinary skill in theart by describing in detail exemplary embodiments thereof with referenceto the accompanying drawings.

FIG. 1 is an exemplary configuration diagram illustrating an informationprocessing system for providing information on a medical deviceaccording to an embodiment of the present disclosure.

FIG. 2 is a block diagram of an internal configuration of an informationprocessing system according to an embodiment of the present disclosure.

FIG. 3 is a functional block diagram of an internal configuration of aprocessor according to an embodiment of the present disclosure.

FIG. 4 is a flowchart illustrating a method for determining anabnormality in a medical device according to an embodiment of thepresent disclosure.

FIG. 5 is a diagram illustrating an example of a first machine learningmodel according to an embodiment of the present disclosure.

FIG. 6 is a diagram illustrating an example of a second machine learningmodel according to an embodiment of the present disclosure.

FIG. 7 is a diagram illustrating an example of a third machine learningmodel according to an embodiment of the present disclosure.

FIG. 8 is a diagram illustrating example display of a medical device ona medical image according to an embodiment of the present disclosure.

FIG. 9 is a diagram illustrating example display of a medical device ona medical image displayed with reference to a fiducial marker accordingto an embodiment of the present disclosure.

FIG. 10 is a flowchart illustrating a training method for determining anabnormality in a medical device according to an embodiment of thepresent disclosure.

FIG. 11 is a diagram illustrating an example of a fourth machinelearning model according to an embodiment of the present disclosure.

FIG. 12 is an exemplary diagram illustrating an artificial neuralnetwork model according to an embodiment of the present disclosure.

FIG. 13 is a diagram illustrating an example of generating training dataaccording to an embodiment of the present disclosure.

FIG. 14 is a block diagram of any computing device associated with thedetermination of an abnormality of a medical device according to anembodiment of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, specific details for the practice of the present disclosurewill be described in detail with reference to the accompanying drawings.However, in the following description, detailed descriptions ofwell-known functions or configurations will be omitted if it may makethe subject matter of the present disclosure rather unclear.

In the accompanying drawings, the same or corresponding components areassigned the same reference numerals. In addition, in the followingdescription of the embodiments, duplicate descriptions of the same orcorresponding components may be omitted. However, even if descriptionsof components are omitted, it is not intended that such components arenot included in any embodiment.

Advantages and features of the disclosed embodiments and methods ofaccomplishing the same will be apparent by referring to embodimentsdescribed below in connection with the accompanying drawings. However,the present disclosure is not limited to the embodiments disclosedbelow, and may be implemented in various forms different from eachother, and the present embodiments are merely provided to make thepresent disclosure complete, and to fully disclose the scope of thedisclosure to those skilled in the art to which the present disclosurepertains.

The terms used herein will be briefly described prior to describing thedisclosed embodiments in detail. The terms used herein have beenselected as general terms which are widely used at present inconsideration of the functions of the present disclosure, and this maybe altered according to the intent of an operator skilled in the art,related practice, or introduction of new technology. In addition, inspecific cases, certain terms may be arbitrarily selected by theapplicant, and the meaning of the terms will be described in detail in acorresponding description of the embodiments. Therefore, the terms usedin the present disclosure should be defined based on the meaning of theterms and the overall content of the present disclosure rather than asimple name of each of the terms.

As used herein, the singular forms “a,” “an,” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesthe singular forms. Further, the plural forms are intended to includethe singular forms as well, unless the context clearly indicates theplural forms. Further, throughout the description, if a portion isstated as “comprising (including)” a component, it intends to mean thatthe portion may additionally comprise (or include or have) anothercomponent, rather than excluding the same, unless specified to thecontrary.

Further, the term “module” or “unit” used herein refers to a software orhardware component, and “module” or “unit” performs certain roles.However, the meaning of the “module” or “unit” is not limited tosoftware or hardware. The “module” or “unit” may be configured to be inan addressable storage medium or configured to play one or moreprocessors. Accordingly, as an example, the “module” or “unit” mayinclude components such as software components, object-oriented softwarecomponents, class components, and task components, and at least one ofprocesses, functions, attributes, procedures, subroutines, program codesegments, drivers, firmware, micro-codes, circuits, data, database, datastructures, tables, arrays, and variables. Furthermore, functionsprovided in the components and the “modules” or “units” may be combinedinto a smaller number of components and “modules” or “units”, or furtherdivided into additional components and “modules” or “units.”

According to an embodiment, the “module” or “unit” may be implemented asa processor and a memory. The “processor” should be interpreted broadlyto encompass a general-purpose processor, a central processing unit(CPU), a microprocessor, a digital signal processor (DSP), a controller,a microcontroller, a state machine, and so forth. Under somecircumstances, the “processor” may refer to an application-specificintegrated circuit (ASIC), a programmable logic device (PLD), afield-programmable gate array (FPGA), and so on. The “processor” mayrefer to a combination for processing devices, e.g., a combination of aDSP and a microprocessor, a combination of a plurality ofmicroprocessors, a combination of one or more microprocessors inconjunction with a DSP core, or any other combination of suchconfigurations. In addition, the “memory” should be interpreted broadlyto encompass any electronic component that is capable of storingelectronic information. The “memory” may refer to various types ofprocessor-readable media such as random access memory (RAM), read-onlymemory (ROM), non-volatile random access memory (NVRAM), programmableread-only memory (PROM), erasable programmable read-only memory (EPROM),electrically erasable PROM (EEPROM), flash memory, magnetic or opticaldata storage, registers, and so on. The memory is said to be inelectronic communication with a processor if the processor can readinformation from and/or write information to the memory. The memoryintegrated with the processor is in electronic communication with theprocessor.

In the present disclosure, a “system” may include at least one of aserver device and a cloud device, but not limited thereto. For example,the system may include one or more server devices. In another example,the system may include one or more cloud devices. In still anotherexample, the system may include both the server device and the clouddevice operated in conjunction with each other.

In the present disclosure, a “display” may refer to any display deviceassociated with a computing device, and for example, it may refer to anydisplay device that is controlled by the computing device, or that candisplay any information/data provided from the computing device.

In the present disclosure, the “artificial neural network model” is anexample of the machine learning model, and may include any model used toinfer an answer to a given input. According to an embodiment, theartificial neural network model may include an artificial neural networkmodel including an input layer, a plurality of hidden layers, and anoutput layer. In an example, each layer may include one or more nodes.In addition, the artificial neural network model may include weightsassociated with a plurality of nodes included in the artificial neuralnetwork model. In an example, the weights may include any parameter thatis associated with the artificial neural network model.

In the present disclosure, a “medical image” may refer to any image,picture, and the like associated with the medical field. In addition,the medical image may refer to an image or a picture obtained bycapturing at least a part of a patient’s body, and may include a 2Dimage, a 3D image, a synthetic image, and the like, captured in the formof X-ray, Computed Tomography (CT), Magnetic Resonance Imaging (MRI),Position Emission Tomography (PET), Single Photon Emission CT (SPECT),Digital Breast Tomosynthesis (DBT), and the like, for example. Inaddition, the “reference medical image” may refer to the medical imagedescribed above and which may be an image used for training any machinelearning model. In the present disclosure, the medical image may referto a reference medical image, or conversely, the reference medical imagemay refer to the medical image.

In the present disclosure, the “annotation” may refer to an annotationwork and/or annotation information (e.g., label information, and thelike) determined by performing the annotation work. In addition, the“annotation information” may refer to information for the annotationwork and/or information (e.g., label information) generated by theannotation work. In the present disclosure, the annotation may beperformed by image-level labeling, pixel-level labeling, or the likefor, included in the medical image, the medical device (position, shape,the presence or absence of damage, and the like of the medical device),a fiducial marker (position, shape, and the like of the fiducialmarker), a normal area (position, shape of the normal area, and thelike) according to the fiducial marker, and the like. For example, theannotation may include a segmentation annotation. In an example, thesegmentation annotation may refer to an annotation technique forclassifying all pixels of an image into corresponding classes and thenclassifying and labeling objects in the image.

In the present disclosure, the “medical device” may refer to any medicalinstrument and device that is inserted into or attached to the body of apatient, and the medical device may include, for example, anendotracheal tube (E-tube), a nasogastric tube (N-tube), a centralvenous catheter, a pulmonary artery catheter (Swan-Ganz catheter), achest tube, a pericardiocentesis tube, a Cardiac implantable electronicdevice (CIED), and the like. In addition, by “at least a part of themedical device,” it may refer to a part of the medical device which isnecessary for the determination of an abnormality in the medical device.In addition, by the “abnormality in the medical device,” it may includethe medical device (or a part of the medical device) being notpositioned in the normal area, the medical device being damaged orbroken, and the like. In an example, the “target medical device” refersto the medical device described above, which may be a medical device asa target of the abnormality determination, and the “reference medicaldevice” may refer to the medical device described above, which may be amedical device included in the reference medical image used for trainingany machine learning model. In the present disclosure, the medicaldevice may refer to the target medical device or the reference medicaldevice, or conversely, the target medical device or the referencemedical device may refer to the medical device.

In the present disclosure, the “normal area” may be an area of idealposition where the medical device should be positioned. For example, thenormal area may be determined differently according to each medicaldevice, anatomy of a body organ of the patient, and the like. Such anormal area may be determined from a fiducial marker, which may bedisplayed on the medical image in the form of, for example, a mask, anarea, a contour, a line, a point, and the like in the medical image.

In the present disclosure, the “fiducial marker” is a marker that servesas a criterion for determining the normal area, and may be displayed inthe form of a mask, an area, a contour, a line, a point, and the like inthe medical image. For example, if the normal area is determined to bean area within a predetermined distance from the carina, the fiducialmarker may be an area corresponding to the carina. In an example, thereference fiducial marker may refer to the fiducial marker describedabove, which may be a fiducial marker used for training any machinelearning model. In the present disclosure, the fiducial marker may referto the reference fiducial marker, or conversely, the reference fiducialmarker may refer to the fiducial marker.

In the present disclosure, “instructions” may refer to one or moreinstructions grouped based on functions, which are the components of acomputer program and executed by the processor.

FIG. 1 is an exemplary configuration diagram illustrating an informationprocessing system 120 for providing information on a medical deviceaccording to an embodiment of the present disclosure. As illustrated,the information processing system 120 may be configured so as to becommunicatively connected to each of a user terminal 130 and a storagesystem 110. While FIG. 1 is a diagram illustrating one user terminal130, embodiments are not limited thereto, and in an exemplaryconfiguration, a plurality of user terminals 130 may be connected to theinformation processing system 120 for communication. In addition, whilethe information processing system 120 is illustrated as one computingdevice in FIG. 1 , embodiments are not limited thereto, and theinformation processing system 120 may be configured to processinformation and/or data in a distributed manner through a plurality ofcomputing devices. In addition, while the storage system 110 isillustrated as a single device in FIG. 1 , embodiments are not limitedthereto, and the system may be configured with a plurality of storagedevices or as a system that supports cloud. In addition, respectivecomponents of the system for providing information on the medical deviceillustrated in FIG. 1 represent functional components that can bedivided on the basis of functions, and in an actual physicalenvironment, a plurality of components may be implemented as beingincorporated with each other.

The storage system 110 is a device or cloud system that stores andmanages various data associated with a machine learning model forproviding information on the medical device included in a medical image150. For efficient data management, the storage system 110 may store andmanage various types of data using a database. In an example, thevarious types of data may include any data associated with the machinelearning model (e.g., weights, parameters, input and output values, andthe like associated with the machine learning model). Furthermore, thedata may include information on the medical device, information on thefiducial marker, information on the normal area, information indicatingwhether or not the medical device is positioned in the normal area,information indicating the presence or absence of damage in the medicaldevice, and the like, but embodiments are not limited thereto. WhileFIG. 1 shows the information processing system 120 and the storagesystem 110 as separate systems, embodiments are not limited thereto, andthey may be incorporated into one system.

The information processing system 120 and/or the user terminal 130 areany computing devices that are used to provide information on themedical device included in the medical image. In an example, thecomputing device may refer to any type of device equipped with acomputing function, and may be a notebook, a desktop, a laptop, a tabletcomputer, a server, a cloud system, and the like, for example, but isnot limited thereto. The information processing system 120 may outputthe medical image 150 to a display device of the user terminal 130 toprovide a user 140 with the same. According to an embodiment of thepresent disclosure, the information processing system 120 may providethe user 140 with, through the user terminal 130, the medical image 150including text, guide lines, indicators, and the like indicating whetheror not the target medical device is included, a fiducial markerassociated with the target medical device, whether or not the targetmedical device is positioned in the normal area, the presence or absenceof damage in the target medical device, and the like.

According to an embodiment, the information processing system 120 mayreceive the medical image 150. In an example, the medical image 150 mayrefer to any image, picture, and the like associated with the medicalfield, and may include a 2D image, a 3D image, a synthetic image, andthe like captured or generated in the form of X-ray, Computed Tomography(CT), Magnetic Resonance Imaging (MRI), Position Emission Tomography(PET), Single Photon Emission CT (SPECT), Digital Breast Tomosynthesis(DBT), and the like, for example. Such a medical image 150 may bedirectly captured by a device associated with the information processingsystem 120, or may be received from an external system (e.g., the userterminal 130, the storage system 110, and the like).

Then, the information processing system 120 may detect a target medicaldevice 152 (or information on at least a part of the target medicaldevice 152) included in the received medical image 150. In an example,the medical device may refer to any medical instrument or device that isinserted into or attached to the body of a patient. In addition, theinformation on the medical device (or information on at least a part ofthe medical device) may include information on a type, a name, and thelike of the medical device that can specify the medical device includedin the medical image 150, and information on a position, a shape, and asize of the specified medical device (or of at least a part of themedical device), the presence or absence of an abnormality in themedical device, the presence or absence of damage in the medical device,and/or a group in which the medical device is included, and the like.For example, the medical device may include an endotracheal tube(E-tube), a nasogastric tube (N-tube), a central venous catheter, apulmonary artery catheter (Swan-Ganz catheter), a chest tube, apericardiocentesis tube, a Cardiac implantable electronic device (CIED),and the like.

The information processing system 120 may extract a normal area 156associated with the target medical device 152 from the received medicalimage 150. For example, the normal area 156 may be determined by afiducial marker 154. In an example, the fiducial marker 154 may be amarker serving as a criterion for determining the normal area 156associated with the target medical device 152, and may include at leastone of a point, a line, and an area. In addition, the normal area 156may be an area of ideal position where the target medical device 152should be positioned. For example, the normal area of the endotrachealtube (the normal area where the tube tip of the endotracheal tube shouldbe positioned) may be an area within the airway positioned about 5 cmabove the keel (carina), which is the fiducial marker, and the normalarea of the nasogastric tube may be a position lower than the diaphragmwhich is the fiducial marker, and the normal area of the central venoustube may be an area ranging from the superior vein to the right atriumof the heart. In addition, the normal area of the pulmonary arterycatheter may be an area within the right or left pulmonary arterybranching from the main pulmonary artery, the normal area of the chesttube may be an area within the chest tube, and the normal area of thepericardial puncture tube may be within the shadow of the heart. Thatis, if the target medical device 152 included in the medical image 150is specified, the information processing system 120 may extract thefiducial marker 154 such as a position (anatomical position of a bodyorgan), a size, and the like of a body organ for determining the normalarea 156 of the target medical device 152, and extract the normal area156 associated with the specific target medical device 152 through thefiducial marker 154 from the medical image 150.

The information processing system 120 may determine an abnormality inthe target medical device 152 based on information on the medicaldevice, the normal area 156, and the like. For example, the abnormalityin the target medical device 152 may include the presence or absence ofdamage in the target medical device 152, whether or not the targetmedical device 152 is positioned on a normal area, and the like.According to an embodiment, the information processing system 120 mayuse a predetermined algorithm, a machine learning model, and the like,to determine whether or not the target medical device 152 is damaged,whether or not the target medical device 152 is positioned on the normalarea, and the like, based on information on the target medical device152, the normal area 156, and the like.

The information processing system 120 may determine whether or not atleast a part of the target medical device 152 is positioned on thenormal area 156. According to an embodiment, the information processingsystem 120 may determine whether or not at least a part of the targetmedical device 152 is positioned on the normal area 156 based on aplurality of predetermined rules. In an example, the plurality ofpredetermined rules may be determined for each medical device. Forexample, a rule associated with an endotracheal tube may be determinedas “the tube tip should be positioned in an area within the airwaypositioned between 5 cm and 7 cm above the keel”. Additional ly oralternatively, the plurality of predetermined rules may be determinedfor each of a plurality of medical devices belonging to the same group.That is, by using the predetermined rule, the information processingsystem 120 may extract information 158 (e.g., text, image, guide line,indicator, and the like) on whether or not the target medical device 152is positioned in the normal area 156, and display the information on themedical image 150. Additionally or alternatively, the informationprocessing system 120 may use any machine learning model to determinewhether or not at least a part of the target medical device 152 ispositioned on the normal area 156.

According to another embodiment, the information processing system 120may use a trained machine learning model to determine the presence orabsence of an abnormality in the target medical device 152 included inthe medical image 150. In other words, the information processing system120 may detect the target medical device 152, and input the medicalimage 150 into the trained machine learning model without extracting thenormal area 156, to determine the presence or absence of an abnormalityin the target medical device 152.

FIG. 1 illustrates that a text (“medical device nonnality”) indicatingthe presence or absence of an abnormality in the target medical device152 is displayed together with a guide line (arrow) on the right side ofthe normal area 156 in the medical image 150, but embodiments are notlimited thereto, and the information 158 on whether or not the targetmedical device 152 is positioned in the normal area 156 may be displayedin any area of the medical image 150. In addition, FIG. 1 illustratesthat a dotted line box indicating the target medical device 152 and thenormal area 156 is displayed on the medical image 150, but the dottedline box indicating the target medical device 152 and the normal area156 may not be displayed and omitted. In addition, FIG. 1 illustratesthat whether or not the target medical device 152 is positioned in thenormal area 156 is displayed on the medical image 150, but embodimentsare not limited thereto, and the presence or absence of damage in themedical device may be displayed in the medical image 150. With such aconfiguration, even for a patient with various complex findings, theuser 140 may easily acquire the information on the presence or absenceof an abnormality in the medical device determined based on the medicalimage 150 through the information processing system 120. In addition,after a procedure of inserting the medical device into a body orattaching it to a body surface of the patient, the user 140 can quicklyand accurately check whether or not the procedure is performedcorrectly.

FIG. 2 is a block diagram of an internal configuration of theinformation processing system 120 according to an embodiment of thepresent disclosure. The information processing system 120 may include amemory 210, a processor 220, a communication module 230, and an inputand output interface 240. As illustrated in FIG. 2 , the informationprocessing system 120 may be configured to communicate informationand/or data through a network by using the communication module 230.

The memory 210 may include any non-transitory computer-readablerecording medium. According to an embodiment, the memory 210 may includea permanent mass storage device such as random access memory (RAM), readonly memory (ROM), disk drive, solid state drive (SSD), flash memory,and so on. In another example, a non-destructive mass storage devicesuch as ROM, SSD, flash memory, disk drive, and so on may be included inthe information processing system 120 as a separate permanent storagedevice that is distinct from the memory. In addition, the memory 210 maystore an operating system and at least one program code (e.g., codeinstalled and driven in the information processing system 120 to detectinformation on at least a part of a medical device, extract a fiducialmarker, extract a normal area, determine whether or not the medicaldevice is positioned in the normal area, determine the presence orabsence of damage in the medical device, and the like).

These software components may be loaded from a computer-readablerecording medium separate from the memory 210. Such a separatecomputer-readable recording medium may include a recording mediumdirectly connectable to the information processing system 120, and mayinclude a computer-readable recording medium such as a floppy drive, adisk, a tape, a DVD/CD-ROM drive, a memory card, and the like, forexample. In another example, the software components may be loaded intothe memory 210 through the communication module 230 rather than thecomputer-readable recording medium. For example, at least one programmay be loaded into the memory 210 based on a computer program (e.g., aprogram for detecting information on at least a part of the medicaldevice, extracting a fiducial marker, extracting a normal area,determining whether or not the medical device is positioned in thenormal area, and the like) which is installed by the files provided bythe developers, or by a file distribution system that distributes aninstallation file of an application through the communication module230.

The processor 220 may be configured to process the commands of thecomputer program by performing basic arithmetic, logic, and input andoutput operations. The commands may be provided to a user terminal (notillustrated) or another external system by the memory 210 or thecommunication module 230. For example, the processor 220 may receive amedical image, and detect information on at least a part of a targetmedical device included in the received medical image. For example, theprocessor 220 may detect information on the position of at least a partof the target medical device in the received medical image. In addition,the processor 220 may extract a fiducial marker associated with thetarget medical device from the received medical image. Then, theprocessor 220 may determine the presence or absence of an abnormality inthe target medical device based on the information on the target medicaldevice and the extracted fiducial marker. In this case, the processor220 may determine a normal area of the target medical device based onthe extracted fiducial marker, and determine whether or not at least apart of the target medical device is positioned in the normal area. Theprocessor 220 may display the information on whether or not the targetmedical device is positioned on the normal area, on the medical image ina predetermined form (e.g., text, image, guide line, indicator, and thelike).

The communication module 230 may provide a configuration or function forthe user terminal (not illustrated) and the information processingsystem 120 to communicate with each other through a network, and mayprovide a configuration or function for the information processingsystem 120 to communicate with an external system (e.g., a separatecloud system). For example, control signals, commands, data, and thelike provided under the control of the processor 220 of the informationprocessing system 120 may be transmitted to the user terminal and/or theexternal system through the communication module 230 and the networkthrough the communication module of the user terminal and/or an externalsystem. For example, the user terminal and/or the external system mayreceive, from the information processing system 120, the information onwhether or not the target medical device is positioned in the normalarea, the information on the presence or absence of damage in the targetmedical device, and the like.

In addition, the input and output interface 240 of the informationprocessing system 120 may be a means for interfacing with a device (notillustrated) for inputting or outputting, which may be connected to theinformation processing system 120 or included in the informationprocessing system 120. In FIG. 2 , the input and output interface 240 isillustrated as a component configured separately from the processor 220,but embodiments are not limited thereto, and the input and outputinterface 240 may be configured to be included in the processor 220. Theinformation processing system 120 may include more components than thoseillustrated in FIG. 2 . Meanwhile, most of the related components maynot necessarily require exact illustration.

The processor 220 of the information processing system 120 may beconfigured to manage, process, and/or store the information and/or datareceived from a plurality of user terminals and/or a plurality ofexternal systems. According to an embodiment, the processor 220 mayreceive the medical image from the user terminal and/or the externalsystem. In this case, the processor 220 may detect the information on atleast a part of the target medical device included in the receivedmedical image.

FIG. 3 is a functional block diagram of an internal configuration of theprocessor 220 according to an embodiment of the present disclosure. Asillustrated, the processor 220 may include a target medical deviceclassifier 310, a target medical device detector 320, a normal areapredictor 330, and the like. In this case, the processor 220 maycommunicate with a database and/or an external device (e.g., a userterminal or an external system) that includes medical images, andreceive a medical image necessary for determining an abnormality in themedical device.

The target medical device classifier 310 may determine whether or not atarget medical device is included in the received medical image.According to an embodiment, the target medical device classifier 310 maydetermine whether or not the medical device included in the receivedmedical image belongs to the same medical device group as the targetmedical device. To this end, the target medical device classifier 310may group a plurality of medical devices for determining abnormalitiesin the medical devices. For example, the target medical deviceclassifier 310 may use a grouping algorithm, a machine learning model,or the like to group medical devices into a group that can be determinedfor the presence or absence of an abnormality based on one criterion.

According to an embodiment, the target medical device classifier 310 mayinclude a machine learning model trained to detect to which group themedical device included in the medical image belongs. In this case, thetarget medical device classifier 310 may be trained to receive aplurality of reference medical images and output a medical device groupto which a reference medical device included in each of the plurality ofreference medical images belongs. In this case, the target medicaldevice classifier 310 may be trained with the annotated training data.For example, the processor 220 may collect a medical image that includesa medical device (or a group to which the medical device belongs) to bedetermined, and a medical image that does not include a medical device(or a group to which the medical device belongs) to be determined. Then,if there is a medical device to be determined in the medical image, theprocessor 220 may perform labeling (e.g., image level labeling, pixellevel labeling, and the like) on the corresponding medical device, andif there is no medical device to be determined, label 0 to performannotation. By using the annotated medical image as described above, theprocessor 220 may train the target medical device classifier 310.

According to an embodiment, if a medical image is received, the targetmedical device classifier 310 may crop or divide only an area in themedical image in which the medical device to be determined is present.Then, the target medical device classifier 310 may also detect thepresence or absence of the target medical device in the divided medicalimage. Additionally or alternatively, the target medical deviceclassifier 310 may detect only a part of the target medical device(e.g., a tube tip, and the like).

The target medical device detector 320 may detect the information on theposition of at least a part of the target medical device in the receivedmedical image. For example, the target medical device detector 320 maydetect the position of the whole or part of the target medical deviceclassified or detected by the target medical device classifier 310.Additionally, the target medical device detector 320 may further detectthe information on the size, the shape, and the like of the targetmedical device. In this case, the position, the size, the shape, and thelike of the target medical device may be the information calculatedbased on the number of pixels in the medical image, or the informationcalculated based on the relative relationship (relative position, size,and the like) with other body organs and the like included in themedical image.

According to an embodiment, the target medical device detector 320 mayinclude a machine learning model trained to detect the information onthe position of at least a part of the target medical device included inthe received medical image. In this case, the target medical devicedetector 320 may be trained with the annotated training data. Forexample, the processor 220 may acquire a plurality of reference medicalimages that include one or more reference medical devices, and acquirean annotation for the position of at least a part of one or morereference medical devices included in the plurality of reference medicalimages. In this case, the target medical device detector 320 may betrained to receive the plurality of reference medical images, and detectthe information on the reference medical devices included in each of theplurality of reference medical images based on the annotation for theposition of at least a part of the one or more reference medicaldevices. For example, the processor 220 may label the position, thesize, the shape, and the like of the medical device (or a part of themedical device) in the medical image including the medical device toperform annotation. In an example, the annotation may be performed inthe form of a mask, an area, a contour, a line, a point, and the like,indicating the information on the medical device. In this case, theannotation may be respectively performed for each type of medical deviceincluded in the medical image, or may be performed irrespective of thetype. Then, the processor 220 may train the target medical devicedetector 320 using the annotated medical image as described above.

The normal area predictor 330 may extract a fiducial marker associatedwith the target medical device from the received medical image. Inaddition, the normal area predictor 330 may determine the normal area ofthe target medical device based on the extracted fiducial marker. Forexample, the normal area predictor 330 may extract a fiducial markerassociated with the medical device classified or detected by the targetmedical device classifier 310. In addition, the normal area predictor330 may determine the normal area based on the extracted fiducialmarker. In this case, the normal area predictor 330 may determine thenormal area from the fiducial marker based on a predetermined criterion(a criterion according to the corresponding medical device or group ofmedical devices). For example, if the endotracheal tube is the detectedmedical device, the normal area predictor 330 may extract a keel(carina) area as a fiducial marker for the endotracheal tube. Then, thenormal area predictor 330 may extract an area in the airway near thearea 5 cm above the keel area as the normal area. That is, the normalarea may be determined differently according to medical devices,positions of body organs of a patient included in the medical image, andthe like. In FIG. 3 , it has been described above that the normal areapredictor 330 extracts the fiducial marker and determines the normalarea based on the fiducial marker, but embodiments are not limitedthereto, and the normal area predictor 330 may be divided into two ormore modules, such as a module for extracting a fiducial marker and amodule for determining a normal area.

According to an embodiment, the normal area predictor 330 may include amachine learning model trained to extract a fiducial marker associatedwith a target medical device from a received medical image. In thiscase, the normal area predictor 330 may be trained with the annotatedtraining data. For example, the processor 220 may acquire a plurality ofreference medical images that include one or more reference medicaldevices, and acquire an annotation for a reference fiducial markerassociated with one or more reference medical devices included in theplurality of reference medical images. That is, the processor 220 maylabel a fiducial marker associated with the medical device in themedical image for each type of medical device or for each group to whicheach medical device belongs and perform annotation. In an example, theannotation may be performed in the form of a mask, an area, a contour, aline, a point, or the like that indicates the fiducial marker. In thiscase, the normal area predictor 330 may be trained to receive theplurality of reference medical images, and extract a reference fiducialmarker associated with the one or more reference medical devices in theplurality of reference medical images based on the annotation for thereference fiducial marker associated with the one or more referencemedical devices. In FIG. 3 , it has been described above that the normalarea predictor 330 first extracts the fiducial marker and thendetermines the normal area associated with the medical device based onthe fiducial marker, but embodiments are not limited thereto, and thenormal area predictor 330 may directly determine, or be trained todetermine, the normal area associated with the medical device withoutfirst extracting the fiducial marker.

According to an embodiment, the processor 220 may determine the presenceor absence of an abnormality, that is, it 220 may determine, forexample, whether or not at least a part of the target medical device ispositioned in the normal area and the like, using the information on theposition of at least a part of the medical device extracted by thetarget medical device detector 320, and the fiducial marker and/or thenormal area extracted by the normal area predictor 330. For example, theprocessor 220 may determine the presence or absence of an abnormalityusing a predetermined rule, or may determine the presence or absence ofan abnormality using a machine learning model. Then, the processor 220may display information associated with the determined presence orabsence of an abnormality in association with the medical image. Forexample, the information associated with the presence or absence of anabnormality may be displayed using text, image, guide line, indicator,and the like.

Additionally or alternatively, training data for training the targetmedical device classifier 310, the target medical device detector 320,and the normal area predictor 330 may be generated by the processor 220.For example, the processor 220 may receive a reference medical imagethat does not include a medical device. In addition, the processor 220may determine, in the reference medical image, a reference medicaldevice for generating training data, and a normal area associated withthe reference medical device. Then, the processor 220 may display atleast a part of the reference medical device in the determined normalarea in the reference medical image to generate a first set of trainingdata in which the reference medical device is normally positioned. Inaddition, the processor 220 may display at least a part of the referencemedical device in an area other than the determined normal area in thereference medical image to generate a second set of training data inwhich the reference medical device is abnormally positioned. In thiscase, the processor 220 may train the target medical device classifier310, the target medical device detector 320 and/or the normal areapredictor 330 based on the generated first set of training data andsecond set of training data. Additionally or alternatively, theprocessor 220 may train any machine learning model that receives themedical image and outputs the presence or absence of an abnormality inthe medical device, based on the generated first set of training dataand second set of training data and . With this configuration, theprocessor 220 can efficiently generate a large amount of training datafor training the machine learning model even in a situation where it isdifficult to collect medical images associated with the medical device.

Although the components of the processor 220 have been describedseparately for each function in FIG. 3 , it does not necessarily meanthat they are physically separated. For example, the target medicaldevice detector 320 and the normal area predictor 330 have beendescribed above as separate components, but this is for betterunderstanding of the disclosure, and embodiments are not limitedthereto. For example, the target medical device classifier 310, thetarget medical device detector 320, and the normal area predictor 330may be implemented through one machine learning model, or may beimplemented through a plurality of different machine learning models. Inaddition, while FIG. 3 illustrates that there is an abnormality in themedical device if the medical device is not positioned on the normalarea, but embodiments are not limited thereto, and it may also bedetermined that there is an abnormality in the medical device if themedical device is damaged.

FIG. 4 is a flowchart illustrating a method 400 for determining anabnormality in the medical device according to an embodiment of thepresent disclosure. According to an embodiment, the method 400 fordetermining an abnormality in the medical device may be performed by aprocessor (e.g., a processor of the user terminal and/or at least oneprocessor of the information processing system). As illustrated, themethod 400 for determining an abnormality in the medical device may beinitiated by the processor receiving a medical image (S410). Forexample, the processor may directly capture the medical images using anydevice associated with the information processing system, or receive themedical images from an external device (e.g., user terminal ordatabase).

The processor may detect the information on at least a part of thetarget medical device included in the received medical image (S420). Inan example, the information on at least a part of the medical device mayinclude a position of the whole or part of the medical device,information on a group to which the medical device belongs, whether ornot the medical device corresponds to the target medical device, and thelike. The processor may use the first machine learning model to detectthe information on the position of at least a part of the target medicaldevice in the received medical image. In this case, the processor mayacquire a plurality of reference medical images that include one or morereference medical devices, and acquire an annotation for the position ofat least a part (e.g., tube tip, whole tube) of the one or morereference medical devices included in the plurality of reference medicalimages. In this case, the first machine learning model may be trained toreceive the plurality of reference medical images, and detect theinformation on the reference medical devices included in each of theplurality of reference medical images based on the annotation for theposition of at least a part of the one or more reference medicaldevices.

According to an embodiment, the processor may use the second machinelearning model to determine whether or not the target medical device isincluded in the received medical image. If the target medical device isincluded in the received medical image, the processor may use the firstmachine learning model to detect the information on the position of atleast a part of the target medical device in the received medical image.In this case, the first machine learning model and the second machinelearning model may be integrated. For example, the processor maydetermine whether or not the medical device included in the receivedmedical image belongs to the same medical device group as the targetmedical device. In an example, the second machine learning model may betrained to receive a plurality of reference medical images and output amedical device group to which the reference medical device included ineach of the plurality of reference medical images belongs.

In order to detect the information on at least a part of the targetmedical device, the processor may extract a fiducial marker associatedwith the target medical device from the received medical image, anddetermine the presence or absence of an abnormality in the targetmedical device based on the information on the target medical device andthe extracted fiducial marker. For example, the presence or absence ofan abnormality in the target medical device may include malposition ofthe target medical device, damage to the target medical device itself,and the like. In this case, the processor may use the third machinelearning model to extract the fiducial marker associated with the targetmedical device from the received medical image. For example, theprocessor may acquire a plurality of reference medical images thatinclude one or more reference medical devices, and acquire an annotationfor the reference fiducial marker associated with one or more referencemedical devices included in the plurality of reference medical images.In this case, the third machine learning model may be trained to receivethe plurality of reference medical images, and extract a referencefiducial markers associated with the one or more reference medicaldevices in the plurality of reference medical images based on theannotation for the reference fiducial marker associated with the one ormore reference medical devices.

FIG. 5 illustrates an example of a first machine learning model 500according to an embodiment of the present disclosure. As illustrated,the first machine learning model 500 may receive a medical image 510 andoutput information 520 on a position of at least a part of a targetmedical device included in the medical image 510. For example, the firstmachine learning model 500 may be accessed by or included in the targetmedical device detector (320 of FIG. 3 ) described above.

According to an embodiment, the first machine learning model 500 may betrained to receive the plurality of reference medical images, and detectthe information on the reference medical devices included in each of theplurality of reference medical images based on the annotation for theposition of at least a part of the one or more reference medicaldevices. For example, in order to generate and train the first machinelearning model 500, the processor (220 of FIG. 2 ) may acquire aplurality of reference medical images that include the reference medicaldevice, and acquire annotation information on the position of at least apart of the reference medical devices included in the plurality ofreference medical images.

Then, the processor may use the information on the position of at leasta part of the annotated reference medical device as ground truth whentraining the first machine learning model 500. For example, theannotation may be performed by image-level labeling, pixel-levellabeling, and the like, and may include segmentation annotation. In anexample, the segmentation annotation may refer to an annotationtechnique for classifying at least some pixels of an image intocorresponding classes and then classifying and labeling the objects inthe image. That is, if the first machine learning model 500 detects atleast a part of the target medical device, the first machine learningmodel 500 may use anatomical segmentation to recognize the determinationof abnormality of the target medical device in and/or outside the body.

Additionally or alternatively, the processor may generate training datafor training the first machine learning model 500 by using a medicalimage that does not include the medical device. For example, theprocessor may determine, in the reference medical image, a referencemedical device for generating training data, and a normal areaassociated with the reference medical device. Then, the processor maydisplay at least a part of the reference medical device in thedetermined normal area in the reference medical image to generate afirst set of training data in which the reference medical device isnormally positioned, and display at least a part of the referencemedical device in an area other than the determined normal area in thereference medical image to generate a second set of training data inwhich the reference medical device is abnormally positioned. In thiscase, the first machine learning model 500 may be trained with thegenerated first set of training data and second set of training data.

Additionally or alternatively, the first machine learning model 500 maybe trained based on information on a lesion included in the referencemedical image. According to an embodiment, if the information on alesion is recognized, the first machine learning model 500 may infer theinformation on the medical device used in relation to the lesion with anincreased accuracy. For example, the first machine learning model 500may be trained to receive a plurality of reference medical images, anddetect the information on the medical device related to a lesionincluded in each of a plurality of reference medical images based on theannotation for one or more lesions (position, size, shape, and the likeof the lesions). For example, in order to generate and train the firstmachine learning model 500, the processor may acquire a plurality ofreference medical images including lesions, and acquire annotations forthe lesions included in the plurality of reference medical images.According to another embodiment, the processor may select a lesiondetection learning model as an initial model for the first machinelearning model 500, and train the first machine learning model 500 toinfer the information on the position of at least a part of the targetmedical device in the medical image. In this way, if the lesiondetection learning model is selected as the initial model for trainingthe first machine learning model 500, a higher accuracy can be providedcompared to when a different model is selected as the initial model.

FIG. 6 illustrates an example of a second machine learning model 600according to an embodiment of the present disclosure. As illustrated,the second machine learning model 600 may receive the medical image 510and determine whether or not the target medical device is included (610)in the received medical image 510. For example, the second machinelearning model 600 may be accessed by or included in the target medicaldevice classifier (310 of FIG. 3 ) described above. In addition, whetheror not the target medical device is included (610) may include whetheror not the medical device included in the medical image 510 belongs tothe same medical device group as the target medical device. Accordingly,the second machine learning model 600 may receive the medical image 510and output information on the groups that the medical device included inthe medical image 510 belongs.

According to an embodiment, the second machine learning model 600 may betrained to receive a plurality of reference medical images and output amedical device group to which a reference medical device included ineach of the plurality of reference medical images belongs. In this case,the second machine learning model 600 may be trained with the annotatedinformation on the medical device group. In an example, the medicaldevice group may include a plurality of medical devices capable ofdetermining the presence or absence of at least a part of the medicaldevice based on the same marker and/or the same normal area.

According to an embodiment, the processor may acquire a plurality ofreference medical images that include one or more reference medicaldevices, and acquire annotations for reference groups associated withone or more reference medical devices included in the plurality ofreference medical images. In this case, the second machine learningmodel 600 may be trained to receive the plurality of reference medicalimages, and extract reference groups associated with the one or morereference medical devices in the plurality of reference medical imagesbased on the annotations for the reference groups associated with theone or more reference medical devices. For example, the annotation maybe performed by image-level labeling, pixel-level labeling, and thelike, and may include segmentation annotation. Then, the processor mayinput the information on the group of reference medical devicesannotated as described above to the second machine learning model 600 totrain the second machine learning model 600.

In FIGS. 5 and 6 , the first machine learning model 500 and the secondmachine learning model 600 have been described above as being separatedfrom each other, but embodiments are not limited thereto. For example,the first machine learning model 500 and the second machine learningmodel 600 may be implemented as one machine learning model or two ormore machine learning models. In another example, the second machinelearning model 600 may be implemented as a plurality of differentmachine learning models generated to extract a plurality of medicaldevice groups.

FIG. 7 illustrates an example of a third machine learning model 700according to an embodiment of the present disclosure. As illustrated,the third machine learning model 700 may receive the medical image 510and extract a fiducial marker 710 associated with a target medicaldevice from the medical image 510. For example, the third machinelearning model 700 may be accessed by or included in the normal areapredictor (330 in FIG. 3 ) described above.

According to an embodiment, the third machine learning model 700 may betrained to output the fiducial marker associated with the target medicaldevice from the medical image. For example, the processor may acquire aplurality of reference medical images that include one or more referencemedical devices, and acquire annotations for the reference fiducialmarkers associated with one or more reference medical devices includedin the plurality of reference medical images. In this case, the thirdmachine learning model 700 may be trained to receive the plurality ofreference medical images, and extract reference fiducial markersassociated with the one or more reference medical devices in theplurality of reference medical images based on the annotations for thereference fiducial markers associated with the one or more referencemedical devices. For example, the annotation may be performed byimage-level labeling, pixel-level labeling, and the like, and mayinclude segmentation annotation.

Additionally or alternatively, the processor may generate training datafor training the third machine learning model 700 by using a referencemedical image that does not include the medical device. As describedabove, the processor may determine or acquire, in the reference medicalimage, a reference medical device for generating training data and anormal area associated with the reference medical device. Then, theprocessor may display at least a part of the medical device in thedetermined normal area in the reference medical image to generate afirst set of training data in which the reference medical device isnormally positioned, and display at least a part of the referencemedical device in an area other than the determined normal area in thereference medical image to generate a second set of training data inwhich the reference medical device is abnormally positioned. In thiscase, the third machine learning model 700 may be trained with thegenerated first set of training data and second set of training data.

In FIGS. 5 to 7 , the first machine learning model 500, the secondmachine learning model 600, and the third machine learning model 700have been described above as being separated from each other, butembodiments are not limited thereto. According to an embodiment, thefirst machine learning model 500, the second machine learning model 600,and the third machine learning model 700 may be implemented as onemachine learning model or implemented as two or more machine learningmodels. For example, one machine learning model may be configured toextract or detect, through MTL (Multi-Task Learning) or the like, theinformation on the position of at least a part of the target medicaldevice, whether or not the target medical device is included, thefiducial markers, and the like. In another example, the third machinelearning model 700 may be implemented as a plurality of machine learningmodels configured to detect the presence or absence of an abnormality ina plurality of target medical devices.

FIG. 8 is a diagram illustrating an example in which medical devices 812and 822 are displayed on medical images 810 and 820 according to anembodiment of the present disclosure. As described above, the processor(e.g., 220 of FIG. 2 ) may detect the information on the target medicaldevices 812 and 822 from the medical images 810 and 820. For example,the information on the target medical devices 812 and 822 may includeany information for determining the presence or absence of anabnormality in the medical device.

According to an embodiment, the medical image 810 may represent an imagein which the target medical device 812 is normally positioned. Forexample, if the target medical device 812 is normally positioned, a text(“medical device normality”), guide lines, and the like indicating thatthe target medical device 812 is normal, may be included in the medicalimage 810. In addition, the medical image 820 may represent an image inwhich the target medical device 822 is not normally positioned. Forexample, if there is an abnormality in the target medical device 822, atext (“medical device abnormality”), guide lines, and the likeindicating that there is an abnormality in the target medical device 822may be included in the medical image 820.

According to an embodiment, if receiving the medical images 810 and 820,the processor may input the received medical images 810 and 820 into onemachine learning model to determine the presence or absence of anabnormality in the target medical devices 812 and 822. For example, theone machine learning model may be a model trained to determine anabnormality in a medical device from a medical image. Additionally oralternatively, the processor may input the received medical images 810and 820 into a plurality of machine learning models to determine thepresence or absence of an abnormality in the target medical devices 812and 822. For example, the plurality of machine learning models may bemodels trained to detect or extract information on the position of atleast a part of the target medical device, whether or not the targetmedical device is included, the fiducial marker, and the like from themedical image.

FIG. 9 is a diagram illustrating an example in which medical devices 912and 922 are displayed on medical images 910 and 920 with reference tofiducial markers 914 and 924 according to an embodiment of the presentdisclosure. As described above, the processor (220 of FIG. 2 ) maydetect information on the target medical devices 912 and 922 from themedical images 910 and 920. For example, the information on the targetmedical devices 912 and 922 may be any information for determining thepresence or absence of an abnormality in the medical device, and mayinclude information on the fiducial markers 914 and 924 (e.g., carina)and the like.

According to an embodiment, the medical image 910 may represent an imagein which the target medical device 912 is normally positioned. Forexample, if the medical device 912 is positioned within a predetermineddistance from the fiducial marker 914, it may be determined to bepositioned in the normal area. As illustrated, the processor may extractor determine an area corresponding to the target medical device 912, anarea corresponding to the fiducial marker 914, a normal area determinedfrom the fiducial marker 914, and text indicating the presence orabsence of an abnormality in the target medical device (“the medicaldevice positioned in the normal area”), and the like, and display thesame on the medical image 910. In this case, the processor may displaythe area described above and the like on the medical image 910 usingtext, image, guide line, indicator, or the like.

According to an embodiment, the medical image 920 may represent an imagein which the target medical device 922 is not normally positioned. Forexample, if the medical device 922 is not positioned within apredetermined distance from the fiducial marker 924, it may bedetermined to have an abnormality. As illustrated, the processor mayextract or determine an area corresponding to the target medical device922, an area corresponding to the fiducial marker 924, a normal areadetermined from the fiducial marker 924, and text indicating thepresence or absence of an abnormality in the target medical device (“themedical device positioned outside the normal area”), and the like, anddisplay the same on the medical image 920.

FIG. 10 is a flowchart illustrating a training method 1000 fordetermining an abnormality in a medical device according to anembodiment of the present disclosure. According to an embodiment, thetraining method 1000 for determining an abnormality in a medical devicemay be performed by a processor (e.g., a processor of a user terminaland/or at least one processor of an information processing system). Asillustrated, in the method 1000 for determining an abnormality in themedical device, the processor may receive a reference medical image(S1010). For example, the processor may receive a reference medicalimage that does not include a medical device, which may be takendirectly by any device associated with the information processingsystem, or may receive a reference medical image from an external device(e.g., a user terminal or a database).

The processor may determine a normal area associated with the referencemedical device in the reference medical image (S1020). According to anembodiment, the processor may receive information on the normal areaassociated with the position of at least a part of the reference medicaldevice from the external device, and apply the normal area associatedwith the position of the at least the part associated with the referencemedical device to the reference medical image. Additionally oralternatively, the processor may receive information on a referencemedical device for generating training data from an external device, andextract the normal area associated with the reference medical device inthe reference medical image based on the received information on thereference medical device and the information on the reference medicalimage.

The processor may generate a first set of training data in which atleast a part of the reference medical device is placed in the determinednormal area in the reference medical image (S 1030). For example, atleast a part of the reference medical device may be displayed in thedetermined normal area in the reference medical image, and as a result,the first set of training data may include the training data, that is,the normal training data in which the reference medical device isnormally positioned. In addition, the processor may generate a secondset of training data in which at least a part of the reference medicaldevice is placed in an area other than the determined normal area in thereference medical image (S1040). For example, at least a part of thereference medical device may be displayed in an area other than thedetermined normal area in the reference medical image, and as a result,the second set of training data may include the training data, that is,the abnormal training data in which the reference medical device isabnormally positioned. Then, the processor may train a fourth machinelearning model for determining the presence or absence of an abnormalityin the reference medical device based on the first set of training dataand the second set of training data (S1050). For example, the processormay receive the medical image, detect the information on at least a partof the target medical device included in the received medical image, anduse the trained machine learning model to determine the presence orabsence of an abnormality in the target medical device based on whetheror not the target medical device is placed in the normal area.

According to an embodiment, the fourth machine learning model mayinclude a binary classification model that is trained to classify thereference medical image into normal data or abnormal data. For example,the binary classification model may be trained to output “1” (normaldata) in response to the input of the first set of training data inwhich the reference medical device is normally positioned, and output“0” (abnormal data) in response to the input of the second set oftraining data in which the reference medical device is abnormallypositioned.

FIG. 11 illustrates an example of a fourth machine learning model 1100according to an embodiment of the present disclosure. As illustrated,the fourth machine learning model 1100 may receive a medical image 1110and determine the presence or absence of an abnormality in a targetmedical device 1120.

According to an embodiment, the processor (220 of FIG. 2 ) may receive areference medical image and determine a normal area associated with thereference medical device in the reference medical image. Then, theprocessor may display at least a part of the reference medical device inthe determined normal area in the reference medical image to generate afirst set of training data in which the reference medical device isnormally positioned. In addition, the processor may display at least apart of the reference medical device in an area other than thedetermined normal area in the reference medical image to generate asecond set of training data in which the reference medical device isabnormally positioned. Furthermore, the processor may train a fourthmachine learning model for determining the presence or absence of anabnormality in the reference medical device based on the first set oftraining data and the second set of training data. In an example, thefourth machine learning model may include a binary classification modelthat is trained to classify the reference medical image into normal dataor abnormal data.

Additionally or alternatively, the processor may receive the referencemedical image, display at least a part of the reference medical devicein the reference medical image to generate third training data in whichthe medical device is normal, and display at least a part of damagedreference medical device in the reference medical image to generatefourth training data in which the medical device is abnormal. Then, theprocessor may train a fourth machine learning model for determining thepresence or absence of an abnormality in the reference medical devicebased on the third set of training data and the fourth set of trainingdata.

According to an embodiment, the processor may receive a medical imageand detect information on at least a part of a target medical deviceincluded in the received medical image. Then, the processor may use thetrained fourth machine learning model to determine the presence orabsence of an abnormality in the target medical device based on whetheror not the target medical device is placed in the normal area.Additionally or alternatively, the processor may use the trained fourthmachine learning model to determine the presence or absence of anabnormality in the target medical device based on the presence orabsence of damage in the target medical device. In an example, if thefourth machine learning model is a binary classification model, themedical image may be classified into normal data or abnormal datathrough the fourth machine learning model. To this end, the fourthmachine learning model may be configured to output information (e.g.,“1” or “0”) indicating that the medical image is normal data or abnormaldata, in response to the input of the medical image.

FIG. 12 is an exemplary diagram illustrating an artificial neuralnetwork model 1200 according to an embodiment of the present disclosure.In machine learning technology and cognitive science, the artificialneural network model 1200 as an example of the machine learning modelrefers to a statistical learning algorithm implemented based on astructure of a biological neural network, or to a structure thatexecutes such algorithm.

According to an embodiment, the artificial neural network model 1200 mayrepresent a machine learning model having a problem solving ability byrepeatedly adjusting the weights of synapses by the nodes that areartificial neurons forming the network through synaptic combinations asin the biological neural networks, thus training to reduce errorsbetween a target output corresponding to a specific input and a deducedoutput. For example, the artificial neural network model 1200 mayinclude any probability model, neural network model, and the like, thatis used in artificial intelligence training methods such as machinelearning and deep learning.

According to an embodiment, the first to fourth machine learning modelsdescribed above may be generated in the form of the artificial neuralnetwork model 1200. For example, the artificial neural network model1200 may receive a medical image and output information on at least apart of a target medical device included in the medical image. Inanother example, the artificial neural network model 1200 may detectinformation on a fiducial marker, a normal area, and the like associatedwith the target medical device included in the medical image. As anotherexample, the artificial neural network model 1200 may extract the normalarea associated with the target medical device from the medical image.In still another example, the artificial neural network model 1200 maydetect, from the received medical image, whether or not the targetmedical device is included.

The artificial neural network model 1200 is implemented as a multilayerperceptron (MLP) formed of multiple nodes and connections between them.The artificial neural network model 1200 according to an embodiment maybe implemented using one of various artificial neural network modelstructures including the MLP. As illustrated in FIG. 12 , the artificialneural network model 1200 includes an input layer 1220 to receive aninput signal or data 1210 from the outside, an output layer 1240 tooutput an output signal or data 1250 corresponding to the input data,and (n) number of hidden layers 1230_1 to 1230_n (where n is a positiveinteger) positioned between the input layer 1220 and the output layer1240 to receive a signal from the input layer 1220, extract thefeatures, and transmit the features to the output layer 1240. In anexample, the output layer 1240 receives signals from the hidden layers1230_1 to 1230_n and outputs them to the outside.

The method of training the artificial neural network model 1200 includesthe supervised learning that trains to optimize for solving a problemwith inputs of teacher signals (correct answers), and the unsupervisedlearning that does not require a teacher signal. The informationprocessing system may train, by supervised and/or unsupervised learning,the artificial neural network model 1200 to determine whether or not atleast a part of one or more medical devices included in the medicalimage is positioned on one or more normal areas. The artificial neuralnetwork model 1200 trained as described above may be stored in a memory(not illustrated) of the information processing system, and maydetermine whether or not at least a part of the target medical deviceincluded in the medical image received from the communication moduleand/or the memory is positioned on the normal area, the presence orabsence of damage in at least a part of the target medical device, andthe like.

According to an embodiment, the information processing system maydirectly generate the training data for training the artificial neuralnetwork model 1200 through simulation. For example, the informationprocessing system may receive a reference medical image. In addition,the information processing system may determine a normal area associatedwith a reference medical device in the reference medical image, anddisplay at least a part of the reference medical device in thedetermined normal area in the reference medical image to generate afirst set of training data in which the reference medical device isnormally positioned. In addition, the information processing system maydisplay at least a part of the reference medical device in an area otherthan the determined normal area in the reference medical image togenerate a second set of training data in which the reference medicaldevice is abnormally positioned. Then, the information processing systemmay train the artificial neural network model 1200 for determining thepresence or absence of an abnormality in the reference medical devicebased on the first set of training data and the second set of trainingdata.

According to an embodiment, the input variable of the artificial neuralnetwork model 1200 may include a medical image associated with a medicaldevice or any information indicating the medical image. Additionally oralternatively, the input variable of the artificial neural network model1200 may include a first set of training data in which the medicaldevice is normally positioned, a second set of training data in whichthe medical device is abnormally positioned, and the like. In addition,when the artificial neural network model 1200 is trained, information onthe position of at least a part of the annotated target medical device,information on a fiducial marker and/or a normal area associated withthe annotated target medical device, and the like may be used as groundtruth.

As described above, if the input variable described above is inputthrough the input layer 1220, the output variable output from the outputlayer 1240 of the artificial neural network model 1200 may be a vectorindicating or characterizing the information on the position of at leasta part of the medical device, the information on whether or not thetarget medical device is included, the fiducial marker associated withthe medical device, and/or the presence or absence of an abnormality inthe target medical. Additionally or alternatively, the output variableoutput from the output layer 1240 of the artificial neural network model1200 may be a vector indicating or characterizing the information on thepresence or absence of damage in at least a part of the target medicaldevice included in the medical image.

As described above, the input layer 1220 and the output layer 1240 ofthe artificial neural network model 1200 are respectively matched with aplurality of output variables corresponding to a plurality of inputvariables, and as the synaptic values between nodes included in theinput layer 1220, and the hidden layers 1230_1 to 1230_n, and the outputlayer 1240 are adjusted, training can be processed to extract a correctoutput corresponding to a specific input. Through this training process,the features hidden in the input variables of the artificial neuralnetwork model 1200 can be confirmed, and the synaptic values (orweights) between the nodes of the artificial neural network model 1200can be adjusted so that there can be a reduced error between the targetoutput and the output variable calculated based on the input variable.By using the artificial neural network model 1200 trained as describedabove, the information on at least a part of the target medical deviceincluded in the received medical image may be output.

FIG. 13 is a diagram illustrating an example of generating training dataaccording to an embodiment of the present disclosure. According to anembodiment, the processor (220 of FIG. 2 ) may use a reference medicalimage 1310 that does not include the medical device, to generatetraining data for determining an abnormality in the medical device. Forexample, the processor may display any reference medical device at aspecific position in the reference medical image 1310 to generate amedical image 1320 in which a medical device is normal and/or a medicalimage 1330 in which a medical device has an abnormality.

According to an embodiment, the processor may receive the referencemedical image 1310, and determine a normal area 1314 associated with thereference medical device in the reference medical image 1310. Forexample, the processor may determine that the medical device forgenerating the training data is an endotracheal tube. In this case, theprocessor may extract a fiducial marker 1312 serving as a determinationcriterion to determine the normal area 1314 of the endotracheal tube.For example, the fiducial marker 1312 for determining the normal area1314 of the endotracheal tube may be a carina. The processor may thenset the area between 5 cm and 7 cm above the carina as the normal area1314 of the endotracheal tube.

According to an embodiment, the processor may display, throughsimulation, at least a part of the reference medical device 1322 in thedetermined normal area 1314 in the reference medical image 1310, to thusgenerate a first set of training data in which the reference medicaldevice 1322 is normally positioned. In the illustrated example, theprocessor may display the tube tip of the endotracheal tube positionedin the normal area 1314, to thus generate training data in which thereference medical device 1322 is normally positioned. FIG. 13illustrates that the reference medical device 1322 is displayed togenerate one training data, but embodiments are not limited thereto, andthe reference medical device 1322 may be displayed such that the tubetip is positioned in the normal area 1314, to generate a plurality oftraining data.

According to an embodiment, the processor may display, throughsimulation, at least a part of a reference medical device 1332 in anarea other than the normal area determined in the reference medicalimage 1310, to thus generate a second set of training data in which thereference medical device 1332 is abnormally positioned. In theillustrated example, the processor may display the tube tip of theendotracheal tube positioned in an area other than the normal area 1314,to thus generate training data in which the reference medical device1332 is abnormally positioned. FIG. 13 illustrates that the referencemedical device 1332 is displayed to generate one training data, butembodiments are not limited thereto, and the reference medical device1332 may be displayed with a part (e.g., the tube tip) of the referencemedical device 1332 positioned or not positioned in the normal area 1314to generate a plurality of training data.

Then, the processor may train at least one machine learning model fordetermining the presence or absence of an abnormality in the referencemedical device based on the generated first set of training data andsecond set of training data. With such a configuration, even when it isdifficult to collect a large amount of medical images in which themedical device normally or abnormally displayed, the processor caneffectively generate the training data for training the artificialneural network model only with the medical images.

FIG. 14 is a block diagram of any computing device 1400 associated withdetermining an abnormality of the medical device according to anembodiment of the present disclosure. For example, the computing device1400 may include the information processing system 120 and/or the userterminal 130. As illustrated, the computing device 1400 may include oneor more processors 1410, a bus 1430, a communication interface 1440, amemory 1420 for loading a computer program 1460 to be executed by theprocessors 1410, and a storage module 1450 for storing the computerprogram 1460. Meanwhile, only the components related to the embodimentare illustrated in FIG. 14 . Accordingly, those of ordinary skill in theart to which the present disclosure pertains will be able to recognizethat other general-purpose components may be further included inaddition to the components illustrated in FIG. 14 .

The processors 1410 control the overall operation of each component ofthe computing device 1400. The processors 1410 may be configured toinclude a central processing unit (CPU), a micro processor unit (MPU), amicro controller unit (MCU), a graphic processing unit (GPU), or anytype of processor well known in the technical field of the presentdisclosure. In addition, the processors 1410 may perform an arithmeticoperation on at least one application or program for executing themethod according to the embodiments of the present disclosure. Thecomputing device 1400 may include one or more processors.

The memory 1420 may store various types of data, commands, and/orinformation. The memory 1420 may load one or more computer programs 1460from the storage module 1450 in order to execute the method/operationaccording to various embodiments of the present disclosure. The memory1420 may be implemented as a volatile memory such as RAM, but thetechnical scope of the present disclosure is not limited thereto.

The bus 1430 may provide a communication function between components ofthe computing device 1400. The bus 1430 may be implemented as varioustypes of buses such as an address bus, a data bus, a control bus, or thelike.

The communication interface 1440 may support wired/wireless Internetcommunication of the computing device 1400. In addition, thecommunication interface 1440 may support various other communicationmethods in addition to the Internet communication. To this end, thecommunication interface 1440 may include a communication module wellknown in the technical field of the present disclosure.

The storage module 1450 may non-temporarily store one or more computerprograms 1460. The storage module 1450 may be configured to include anonvolatile memory such as a read only memory (ROM), an erasableprogrammable ROM (EPROM), an electrically erasable programmable ROM(EEPROM), a flash memory, and the like, a hard disk, a detachable disk,or any type of computer-readable recording medium well known in the artto which the present disclosure pertains.

The computer program 1460 may include one or more instructions that, ifloaded into the memory 1420, cause the processors 1410 to perform anoperation/method in accordance with various embodiments of the presentdisclosure. That is, the processors 1410 may perform operations/methodsaccording to various embodiments of the present disclosure by executingone or more instructions.

For example, the computer program 1460 may include instructions forreceiving a medical image and detecting information on at least a partof a target medical device included in the received medical image. Inaddition, the computer program 1460 may include instructions forreceiving a reference medical image, determining a normal areaassociated with a reference medical device in the reference medicalimage, displaying at least a part of the reference medical device in thenormal area determined in the reference medical image to generate afirst set of training data in which the reference medical device isnormally positioned, generating at least a part of the reference medicaldevice in an area other than the determined normal area in the referencemedical image to generate a second set of training data in which thereference medical device is abnormally positioned, and training amachine learning model for determining the presence or absence of anabnormality in the reference medical device based on the first set oftraining data and the second set of training data.

The above description of the present disclosure is provided to enablethose skilled in the art to make or use the present disclosure. Variousmodifications of the present disclosure will be readily apparent tothose skilled in the art, and the general principles defined herein maybe applied to various modifications without departing from the spirit orscope of the present disclosure. Thus, the present disclosure is notintended to be limited to the examples described herein but is intendedto be accorded the broadest scope consistent with the principles andnovel features disclosed herein.

Although example implementations may refer to utilizing aspects of thepresently disclosed subject matter in the context of one or morestandalone computer systems, the subject matter is not so limited, andthey may be implemented in conjunction with any computing environment,such as a network or distributed computing environment. Furthermore,aspects of the presently disclosed subject matter may be implemented inor across a plurality of processing chips or devices, and storage may besimilarly influenced across a plurality of devices. Such devices mayinclude PCs, network servers, and handheld devices.

Although the present disclosure has been described in connection withcertain embodiments herein, it should be understood that variousmodifications and changes can be made without departing from the scopeof the present disclosure, which can be understood by those skilled inthe art to which the present disclosure pertains. Further, suchmodifications and changes are intended to fall within the scope of theclaims appended herein.

What is claimed is:
 1. A computing device, comprising: a memory storingone or more instructions; and a processor configured to, by executingthe stored one or more instructions, receive a medical image, detectinformation on at least a part of a medical device included in thereceived medical image, extract a fiducial marker indicative of ananatomical region associated with the at least the part of medicaldevice from the received medical image, and determine whether themedical device is abnormal based on the extracted fiducial marker. 2.The computing device according to claim 1, wherein the detectedinformation on the at least the part of the medical device includesinformation on a type of the medical device, and wherein the fiducialmarker is extracted based on the type of the medical device.
 3. Thecomputing device according to claim 1, wherein the processor is furtherconfigured to provide information on whether the medical device isabnormal, in association with the medical image.
 4. The computing deviceaccording to claim 3, wherein the providing the information on whetherthe medical device is abnormal includes outputting a graphical markerindicative of the at least the part of the detected medical device andthe extracted fiducial marker on the medical image.
 5. The computingdevice according to claim 3, wherein the providing the information onwhether the medical device is abnormal includes outputting informationon a location of the at least the part of the medical device relative tothe extracted fiducial marker on the medical image.
 6. The computingdevice according to claim 3, wherein the information on whether themedical device is abnormal includes at least one of a position, a shape,a size, a presence or an absence of damage of the medical device, orinformation about a group to which the medical device belongs.
 7. Thecomputing device according to claim 1, wherein the determining whetherthe medical device is abnormal includes determining a normal areaindicative of a region where the medical device is properly located,based on the extracted fiducial marker, and determining whether themedical device is abnormal based on the determined normal area and theextracted fiducial marker.
 8. The computing device according to claim 7,wherein the determining whether the medical device is abnormal based onthe determined normal area and the extracted fiducial marker includes,determining whether the medical device is abnormal based on whether themedical device or a tube tip of the medical device in the receivedmedical image is located in the determined normal area.
 9. The computingdevice according to claim 7, wherein the processor is further configuredto output at least one of the determined normal area or the fiducialmarker on the medical image, and wherein the at least one of the normalarea or the fiducial marker are displayed on the medical image in a formof at least one of a mask, a region, a contour, a line or a dot.
 10. Thecomputing device according to claim 9, wherein a range indicative of thenormal area relative to the fiducial marker is displayed on the medicalimage using a distance indicator.
 11. The computing device according toclaim 7, wherein the processor is further configured to generate thedetermined information on whether the medical device is abnormal basedon the normal area and the fiducial marker, and output the generatedinformation on whether the medical device is abnormal in associationwith the medical image.
 12. The computing device according to claim 11,wherein the outputting the generated information on whether the medicaldevice is abnormal in association with the medical image includes,outputting the information on whether the medical device is abnormal ina form of at least one of text, an image, a guideline or an indicator,in association with the medical image.
 13. A method for determining anabnormality in a medical device in a medical image, the method beingexecuted by at least one of processor and comprising: receiving amedical image, detecting information on at least a part of a medicaldevice included in the received medical image, extracting a fiducialmarker indicative of an anatomical region associated with the medicaldevice from the received medical image, and determining whether themedical device is abnormal based on the extracted fiducial marker.
 14. Anon-transitory, computer-readable medium, the computer-readable mediumcomprising processor-executable code that when executed by a processor,causes the processor to: receive a medical image, detect information onat least a part of a medical device included in the received medicalimage, extract a fiducial marker indicative of an anatomical regionassociated with the medical device from the received medical image, anddetermine whether the medical device is abnormal based on the extractedfiducial marker.